Introduction

• Expert surveys are a key method for measuring complex political concepts, such as political party ideologies (Marks et al. 2007).
• Recent datasets on political party policy positions include over 1,000 observations and > 150 countries, e.g. V-Party (Lührmann et al. 2020) and GPS (Norris 2020).
• To what extent broader coverage in expert surveys comes at the cost of reduced precision?

Objectives

• Examine recent datasets on political party policy positioning: CHES, V-Party, and GPS (Bakker et al. 2020; Lührmann et al. 2020; Norris 2020).
• Investigate whether experts exhibit greater disagreement when evaluating parties: a) in less democratic institutions; b) relatively small; c) policy dimensions (not here).
• Contribute to the literature on expert survey methodologies.

Expert surveys (I)

Experts are individuals with specialized knowledge of one or more countries who can synthesize multiple sources of information when locating individual parties on a policy or ideological scale (Hooghe et al. 2010).

• Cross-national party datasets have expanded from limited geographic and case coverage (Castles and Mair 1984) to nearly global reach (Norris 2020).
• Currently, similar multi-rater design (Maestas 2018) and aggregation methods (Graefe et al. 2014).
• Variation across question wordings, response scales, and pools of experts.

Expert surveys (II)

Expert surveys face two primary types of error: systematic and random error (Maestas 2018)

• Systematic (bias / accuracy): Non-independent assessments. Consistent in the experts’ responses, leading to a deviation from the true estimate. Very problematic because it skews the results.
• Random (noise / precision): Independent, but different assessments. Variability in the experts’ responses that does not follow a specific pattern. Unproblematic if truly independent, but in most cases, the degree of independence is unknown.

Expert surveys (III)

Sources of random error (Budge 2000; Martínez-Coma and Van Ham 2015):

• Individual cognitive and judgment biases (e.g. experts might rely in different heuristics, have ideological leanings or vary in their knowledge).

• Political and social context under which the evaluation is made (e.g. autocracies can punish certain political orientations).

• Some concepts, due its nature, are inherently complex (e.g. political ideology, government effectiveness, political polarization).

Theoretical arguments (I)

Regarding context, random biases have been extensively explored in Europe (Bakker et al. 2014), but less outside of it. We can expect that experts can face information asymmetries depending on the type of political regime:

• Democracies more information, so more precise assessments.
• Autocracies less information, so less precise assessments.

Theoretical arguments (II)

About the unit, some studies have explored the relationship between party size and experts’ disagreement, finding a non-significant relationship (Marks et al. 2007; Steenbergen and Marks 2007). But two competing forces:

• Large parties tend to adopt catch-all strategies, being more ambiguous and difficult to evaluate with precision.
• Small parties tend to lack resources and media attention, being more difficult to evaluate with precision.
• Thus, more precision in “medium” parties?

Theoretical arguments (III)

But though small party units may lack of resources, some can be easier to evaluate if have enough stability and longevity in the party system. One can difference between:

• Statewide small parties, often with shorter lifecycles, being more difficult to evaluate with precision.
• Regional (small) parties, typically tied to specific regionally concentrated groups, more stable, being easier to evaluate.

Theoretical arguments (III)

About the policy dimension, not yet.

Methodology (I)

• Data from 163 countries and 10,041 party-policy observations derived from the V-Party(Lührmann et al. 2020), CHES (Bakker et al. 2020) and GPS (Norris 2020).
• Similar time period, aggregation, and dimensions (next).
• Excluded assessments conducted with only one expert (from 2 to 93).
• Controling for the fixed effects of the dataset.

Methodology (II)

• Dependent variable: Disagreement among experts, using the standard deviation, a common measure of uncertainty (Lührmann et al. 2020; Bakker et al. 2020; Coppedge et al. 2021).

Methodology (III)

• Party size (IV1): Operationalized with the percentage of votes received by the party in the national parliament (Marks et al. 2007; Steenbergen and Marks 2007).

Methodology (IV)

• Regional party (IV2): Defined as political parties that seek representation at the substate level (93 obs.). Data from Massetti and Schakel (2021) and Nordsieck (2019).

Figure 4: State and non-state wide parties

Methodology (V)

• Democracy (IV3): Liberal democracy (continuous) from V-Dem (Coppedge et al. 2021) and dichotomous indicator from DCD (Boix, Miller, and Rosato 2013).

Figure 5: Continuous (V-Dem) and dichotomous (DCD) measures of democracy

Methodology (VII)

• Controls: Expertise is calculated with the ratio of NAs per party-question (Martínez-Coma and Van Ham 2015).

Figure 6: Pairwise correlations grid

Results (I): Bivariate analysis

• Regime type and SD: Slightly negative, seemingly inverted U-curved relationship (specially for small parties).

Results (II): Bivariate analysis

• Size and SD: Slight U-curved association, irrespective of the regime. Less disagreement for medium-sized parties.

Results (III): Bivariate analysis

• Regional and SD: High disagreement on statewide parties, much less on regional parties, specially in autocracies.

Conclusion

• Experts are more precise in democracies, though this can be potentially attributed to higher levels of economic development.
• Democracies and autocracies seem to follow very different patterns, which need to be explored in depth.
• If anything, party size follows a non-linear relationship with noise.
• Experts show minimal disagreement when locating regional (small) parties, ceteris paribus, especially in comparison to small statewide parties.

Work to do

• Explore non-linear relationships (party size and regime type).
• Consider new controls: member of government, internal dissent, or policy-dimension FE.
• Look at variations across policy dimensions.